Exploring User Satisfaction in a Tutorial Dialogue System
نویسندگان
چکیده
User satisfaction is a common evaluation metric in task-oriented dialogue systems, whereas tutorial dialogue systems are often evaluated in terms of student learning gain. However, user satisfaction is also important for such systems, since it may predict technology acceptance. We present a detailed satisfaction questionnaire used in evaluating the BEETLE II system (REVU-NL), and explore the underlying components of user satisfaction using factor analysis. We demonstrate interesting patterns of interaction between interpretation quality, satisfaction and the dialogue policy, highlighting the importance of more finegrained evaluation of user satisfaction.
منابع مشابه
A Tutorial Dialogue System for Real-Time Evaluation of Unsupervised Dialogue Act Classifiers: Exploring System Outcomes
Dialogue act classification is an important step in understanding students’ utterances within tutorial dialogue systems. Machinelearned models of dialogue act classification hold great promise, and among these, unsupervised dialogue act classifiers have the great benefit of eliminating the human dialogue act annotation effort required to label corpora. In contrast to traditional evaluation appr...
متن کاملBeetle II: an adaptable tutorial dialogue system
We present BEETLE II, a tutorial dialogue system which accepts unrestricted language input and supports experimentation with different dialogue strategies. Our first system evaluation compared two dialogue policies. The resulting corpus was used to study the impact of different tutoring and error recovery strategies on user satisfaction and student interaction style. It can also be used in the ...
متن کاملFrom novice to expert: the effect of tutorials on user expertise with spoken dialogue systems
One of the challenges for the current state of the art in spoken dialogue systems is how to make the limitations of the system apparent to users. These limitations have many sources: limited vocabulary, limited grammar, or limitations in the application domain. This study explored the use of a 4-minute tutorial session to acquaint novice users with the features of a spoken dialogue system for a...
متن کاملThe Impact of Interpretation Problems on Tutorial Dialogue
Supporting natural language input may improve learning in intelligent tutoring systems. However, interpretation errors are unavoidable and require an effective recovery policy. We describe an evaluation of an error recovery policy in the BEETLE II tutorial dialogue system and discuss how different types of interpretation problems affect learning gain and user satisfaction. In particular, the pr...
متن کاملTutorial Dialogue in DiBEx
A dialogue-based explanation facility for Intelligent CALL (ICALL) Systems is introduced. Our prototype system, DiBEx, uses meta reasoning to build up an explanation (error) tree, given a faulty user input. It relies on correct grammar sub theories, instead of explicit error taxonomies. The system enters in a tutorial dialogue with the student, where each explanation step is based on the princi...
متن کامل